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ABSTRACT 


Computational experimentation is an important tool of the military. It 
provides useful insights at a lower cost of time and money, when compared to 
physical experiments. Consequently, computational experiments are used to 
evaluate weapon systems for technology acquisition, examine tactics, and to 
help select among alternatives for military operations and war plans. 

Experiments often consist of a large number of factors. Advancements in 
computing power and design of experiments (DOE) for simulation allow for the 
investigation of more of these factors through computational experiments, 
achieved with less expense in time, effort, and money. Within the framework of 
DOE, this thesis investigates Orthogonal Latin Hypercube (OLH) and Nearly 
Orthogonal Latin Hypercube (NOLH) designs. These designs are often used for 
computational experiments. This research greatly expands upon the size (in 
terms of runs and, especially, variables) of the available OLH and NOLH designs. 
Previously, the largest catalogued OLH and NOLH designs were a maximum of 
29 variables and 257 runs. OLH and good space-filling NOLH designs for up to 
512 variables in 1025 runs are now available. This thesis also develops an 
algorithm for handling discrete factors with the designs. Finally, the effects of 
stacking multiple OLH designs into one larger design are quantified. All of the 
designs developed in this research are available at the Simulation Experiments & 
Efficient Designs (SEED) Center website (http://narvest.nps.edu). 
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EXECUTIVE SUMMARY 


The Simulation Experiments & Efficient Designs (SEED) Center of the 
Naval Postgraduate School (NPS) has a mission to “promote the advancement of 
collaborative development and the use of simulation and efficient designs to 
provide decision makers with timely insights on complex systems and 


operations.” 


One of the most widely used designs for computational experiments at the 
SEED Center and elsewhere is the Latin Hypercube (LH) design. LH designs 
are attractive for computational experiments because they allow analysts to 
efficiently vary many factors simultaneously. Furthermore, LH designs provide a 


lot of flexibility for subsequent analysis. 


For any given combination of input factors and runs, there exist a large 
number of possible LH designs. From among the possibilities, LH designs that 
are orthogonal and with good space-filling properties are preferred. Designs with 
good orthogonal properties are desirable because they provide uncorrelated 
estimates for the regression coefficients that analysts frequently use to analyze 
simulation models. Good space-filling properties are useful for data analysis 
across the possible range of data because the experimental points are scattered 
throughout the domain in a uniform-like manner, providing information throughout 
the design space. LH designs which are orthogonal can have poor space-filling 
properties. Thus, a special class of LH designs, called Nearly Orthogonal Latin 
Hypercube (NOLH), which are nearly orthogonal but with improved space-filling 
properties, were developed by Cioppa (2002). 


Existing catalogued Orthogonal Latin Hypercube (OLH) and NOLH 
designs can handle up to 29 factors in 257 runs. In this thesis, the number and 
size of designs currently used by the SEED Center have been dramatically 
increased. Specifically, the readily available OLH and NOLH designs have been 


XV 


increased to 512 input variables and 1025 design points. Table A shows how 
this increases from the previous works of Ye (1998) and Cioppa (2002). 





























m N k (Ye) k (Cioppa) k (Ang) 
4 17 6 7 8 

5 33 8 11 16 

6 65 10 16 32 

7 129 12 22 64 

8 257 14 29 128 

9 513 16 37 256 
10 1025 18 46 512 




















Table A. Maximum number of input variables for OLH and NOLH 
designs from Ye (1998), Cioppa (2002), and Ang (2006). 


By comparing the properties of LH designs with different construction, 
some of the findings include: 


- Florian’s method, used in the construction of NOLH designs, 


improves the orthogonal properties of the LH designs. 


- Increasing the combinations of permutation matrices, to increase 
the number of factors with the same number of runs, tends to cause 


downgraded orthogonal and space-filling properties. 


LH designs assume that the input variables are continuous. The use of 
discrete variables affects the orthogonal and space-filling properties of the 
designs. This thesis provides a useful guide for choosing columns for 
incorporating discrete variables into the catalogued OLH and NOLH designs. 
This guide provides designs with the best orthogonal and space-filling properties. 
Some of the findings include: 


- Positive-Negative coding is best suited for 2-level variables. 


- The orthogonal properties are affected for discrete variables with 


fewer levels, when compared to discrete variable with more levels. 


xvi 


One efficient way to improve the space-filling of an LH design is to stack 
several designs together. While this has been done in the past, the effects have 
not been quantified. This thesis quantifies the improvement of the orthogonal 
and space-filling properties, in relation to the number of stacks, for the various LH 
designs. The LH generator developed in the thesis allows for the flexible creation 


of whatever stacked designs a user desires. 


Users of the LH designs can benefit from the larger designs developed, 
and a clearer understanding of the effects of discrete variables and stacking on 
the orthogonal and space-filling properties of the LH designs. These designs 
and the JAVA generator needed to construct new OLH and NOLH designs are 
available at the SEED Center website (http://harvest.nps.edu). 


xvii 
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I. INTRODUCTION 


A. FRAMING THE PROBLEM 

Computer simulation and experimentation is important to the military. It is 
used in the evaluation of weapon systems and technology prior to acquisition, 
providing the ability to analyze boundary and dangerous conditions that are 
difficult to replicate or are hazardous in physical experiments. Computer 
simulation and experimentation of military operations and war plans, with military 
simulation models like JCATS! and MANA2, allow military decision makers to 
study the effects of tactics, employment options, mix of forces, weapon 
acquisition, and testing of war plans. Through computer simulation and 


experimentation, there are also savings in material, time and money. 


Military uses of computer simulation and experimentation range from the 
acquisition of the 5.56 mm round to the testing of war plans, and each simulation 
or experiment contains numerous factors. These inputs tend to be uncertain 
(i.e., the performance of humans and systems in battle) and it is unreliable to use 
assumptions of strong prior knowledge (Cioppa & Lucas, 2006). Unfortunately, 
the total enumeration of all possible scenarios for experimentation is an 


exhaustive task even with today’s computing power. 


In the field of design of experiments (DOE), there are effective and 
efficient methods to design computational experiments. Specific DOE for 
computer simulation and experimentation, an extension from the traditional DOE, 


has been proposed (Kleijnen, Sanchez, Lucas, & Cioppa, 2005). 


Nearly Orthogonal Latin Hypercube (NOLH) designs possess 
improvements in space-filling properties with nearly orthogonal properties for 
Latin Hypercube (LH) designs (Cioppa, 2006), (Cioppa & Lucas, 2006). NOLH 


designs have been commonly used for military simulation experiments since their 





1 Joint Conflict and Tactical Simulator (JCATS) http://www.jfcom.mil/about/fact_jcats.htm 
(accessed 27 May 2006). 


2 Map Aware Non-uniform Automata (MANA), developed for the New Zealand Army and 
Defense Force. 
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inception (See SEED Center website at http://harvest.nps.edu for more details). 
Current designs readily available include NOLH designs that can handle up to 29 
factors with 257 runs. 


This thesis extends the catalogue of the designs up to 512 factors with 
1025 runs. The effects of using discrete variables with LH designs are also 
investigated, and methods for obtaining LH designs with good orthogonal and 
space-filling properties are proposed. The effects of stacking the LH designs on 
the orthogonal and space-filling properties of the designs are also investigated 
and quantified. 


B. EXPERIMENTS 

By conducting experiments through computer simulation and 
experimentation, one can explore a much greater number of factors than is 
feasible through traditional physical experimentation. Not only does the use of 
computer experimentation provide more consistent replications, but it is a great 


deal more cost efficient than traditional methods of physical experimentation. 


Computer simulation and experimentation are used to compare the 
performance of systems. With traditional experiments, there are numerous 
methods developed to detect input variables that may or may not cause some of 
the variation in the responses. This includes randomization, blocking, and 
replication. In contrast, the use of computer experiments removes the 
requirement for these three methods, as computer experiments provide identical 
answers for repeated runs with the same set of inputs, reducing the need for 
randomization, blocking, and replication (Santer, Williams, & Notz , 2003). 


An experiment consists of responses due to two main types of variables, 
controlled variables and environmental variables. Controlled variables are 
variables that can be set to “control” the product or process. Environmental 
variables are the “uncontrollable” conditions under which the product or process 
is executed. An example of a military experiment is the determination of the 
accuracy of a tank gun: the controlled variables include the weapon system, 
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ammunition type, target types and the operating crew; the uncontrolled variables 
include the weather conditions. The response of the experiment is the distance 
of the impact point from the aim point. 


A metamodel is often used to estimate the effects of the input variables on 
the responses. The utility of metamodels is that they allow the investigation to 
focus on the important factors that may influence the experiment. They are then 
used as quick turn-around estimates of the actual problem and to replace running 
the actual simulation, which generally takes a long time to generate results. A 
description of metamodels can be found in subsection 1. 


The DOE Toolkit (Kleijnen, Sanchez, Lucas, & Cioppa, 2005), which 
provides designs that are used to investigate computer simulation and 


experiments, can be found in summary in subsection 2. 


1. Metamodel 
A metamodel describes the relationship between the output response 


(often denoted as y) and the k input variables (often denoted as x,,x,,...,X,). A 
metamodel is a function g that can be estimated given an experimental design 
and the corresponding responses, as described in Equation 1.1. 
yj = Q(X, Xq5-+) Xx) (1.1) 
The multiple regression model® is often used as the metamodel for 
exploration, where the inputs for the regression coefficients 4,,4,,...,8, are 


usually estimated from computer simulation and experiments by using the 
method of least squares. The error term, ¢, is often assumed to be independent 


and identically distributed (i./.d.) with expectation, E(¢é)=0, and variance, 
Var(e)=07. 


The multiple regression model with k predictor variables is 


k iid 
Q(X. X,)= By + BX + where ¢ ~ Normal(0,o7) (1.2) 
i= 


3 Montgomery, et al. (2001), Chapter 3. 


Metamodels with more complex structures, such as the cubic polynomial 


model, y = 8, + £,x + B,X° + B,x° +e can be represented by Equation 1.2. This is 
achieved by the assignments x, = x, x, = x*, and x, = x°. Similarly, models with 
interactions, i.e., y = 6, + £,X,+ 6X, + B,X,X, +e, and quadratic polynomial with 
interactions models, i.e., y =f + £,X,+ oX + BX, + BypX,X, +e, can also be 


represented by Equation 1.2. 

2. Design of Experiments 

The field of DOE has a rich history in a range of fields, from farming to 
medicine, and provides for the efficient conduct of experiments. Computer 
simulation and experimentation provide a vast area in which to expand DOE. 
Some of the possible designs are proposed by the DOE Toolkit (Kleijnen, 
Sanchez, Lucas, & Cioppa, 2005) shown in Figure 1: 


Response Surface Complexity 











maximal minimal 
screening assump tions 
eee pects istarder 2nd arder smooth en sniod 
iid errars complex errors 
eesti folded SB Very efficient 
Many sequential CSB-X RS FF Latin hypercube 
bifurcation aia (LH) 
(CSB, SB) 
combined CSFD 
ra designs : NOLH 
° 
3 (nearly) saturated frequency designs 
a (2°? FF) 
he 
2 PlackettBurman differential grids 
Z 
R4 
central 
coarse grids ; composite fine grids 
Few Q2* factorial) ed (CCD) (m¥ factorial) 
Figure 1. Recommended designs from DOE Toolkit (Kleiinen, Sanchez, Lucas, 


& Cioppa, 2005). 





The number of factors, along the vertical axis, reflects the number of 
possible “controlled” factors in the simulation experiment. Typical simulation 
experiments consist of large numbers of potential input factors. The toolkit 
describes different designs for different applications. 


The response-surface complexity, along the horizontal axis, defines the 
continuum from simple to complex response surfaces, which are affected by the 
assumptions on the complexity of the metamodel. “Maximal screening” is used 
when the assumed metamodel is simple, i.e., with no interactions or polynomial 
terms, and /./.d. errors. “Minimal assumptions” allow for more complex 
metamodels. The horizontal axis lists these assumptions as guidance for the 
choice of the designs that can be used. 


LH designs are recommended for simulation experiments with large 
numbers of factors and minimal assumptions. For the assumption of a linear 
model, Ye developed an algorithm (Ye, 1998) for orthogonal LH designs which 
were subsequently extended by Cioppa, who exchanged small amounts of non- 
orthogonality for better space-filling and developed the NOLH designs (Cioppa, 
2002). This thesis furthers the current research on these OLH and NOLH 


designs. 


C. RESEARCH GOALS 

Construction of the OLH and NOLH designs requires a large amount of 
computation, computer memory, and time. There are currently a limited number 
of catalogued designs, and this thesis greatly expands the library of the OLH and 


NOLH designs to accommodate greater numbers of variables and runs. 


Traditional LH designs have been used with the assumption of continuous 
variables. However, real world variables often do not take on continuous values, 
rather, they take on discrete values. The orthogonal and space-filling properties 
of the OLH and NOLH designs are diminished when used with discrete variables. 
The thesis explores the effects of this, and provides a general rule of thumb 
when using the LH designs for different numbers and levels of discrete variables. 


D. SOFTWARE 

JAVA is used as the programming language to generate the LH designs in 
this thesis. The code was initially developed to be run within the netBeans IDE 
5.04 running J2SE 5.0. During the course of the development, Eclipse SDK 3.2 
for Windows was subsequently used. In addition, an open source COLT 
package was used to supplement the existing JAVA packages. COLT provides a 
set of Open Source Libraries for High Performance Scientific and Technical 


Computing in Java.® 
With the COLT package, the License Agreement is: 


Copyright (c) 1999 CERN - European Organization for Nuclear 
Research. 


Permission to use, copy, modify, distribute and sell this software 
and its documentation for any purpose is hereby granted without 
fee, provided that the above copyright notice appear in all copies 
and that both that copyright notice and this permission notice 
appear in supporting documentation. CERN makes no 
representations about the suitability of this software for any 
purpose. It is provided "as is" without expressed or implied 
warranty. 


E. THESIS ORGANIZATION 

This thesis extends previous work on OLH and NOLH designs. Chapter II 
focuses on the DOE toolkit, in particular, the orthogonal and space-filling 
properties of LH designs, and the construction of OLH and NOLH designs. 
Chapter Ill investigates the properties of the LH designs with respect to 
categorical and mixed-level variables. Chapter IV covers the results from the 
research. Chapter V concludes and recommends future research. 





4 netBeans 5.0 and J2SE5.0 Update 6 is available from Sun 
http://java.sun.com/j2se/1.5.0/download.jsp (last accessed 29 May 2006). 


5 COLT package from http://dsd.lbl.gov/~hoschek/colt/ (last accessed 29 May 2006). 
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ll. DESIGN OF EXPERIMENTS FOR COMPUTER SIMULATION 
AND EXPERIMENTATION 


A. LATIN HYPERCUBE DESIGN MATRIX 

A design matrix, X, specifies how an experiment is to be conducted, as 
illustrated in Figure 2. The rows of the matrix, n, represent the different design 
points, or runs, for the experiment. Each row represents a single run of the 
experiment. The columns, k, represent the different factors or controlled 
variables for the experiment. The nxk design matrix is used as inputs for the 
experiment, and the responses of the experiment are recorded as the experiment 


Occurs. 


The values in the matrix, represented by (row, column), state the value for 
a specific variable during a specific run of the experiment. For example, the 
representation (7, x7) reflects that the value of variable x7 for run 1 is set at one. 


The notation we use to represent a design matrix is X;’. 


—— k ——> 
variables 


run x1 x2 x3 





1 
2 
3 
A 4 = X run 
5 1 
6 2 
7 3 
8 4 
Figure 2. Design matrices: (a) A 3-factor, 2-level Full-Factorial Design; (b) A 2- 


factor Latin Hypercube Design. 


A Latin Hypercube (LH) design is represented in Figure 2(b). The 
structure of LH designs require the columns to be permutations of the integers 


from {1,2,...,9}, as shown by Xin Figure 2(b). 


B. LATIN HYPERCUBE SAMPLING 

Latin Hypercube Sampling (LHS) is a statistical sampling method to 
generate inputs for computer experiments (McKay, et al., 1979). The inputs for 
the computer experiments can be represented as either a Latin Square for two 


dimensions or an LH design for an arbitrary number of dimensions. 


Orthogonal Latin Hypercube (OLH) and Nearly Orthogonal Latin 
Hypercube (NOLH) designs are special classes of LH designs. OLH designs 
have strict orthogonal properties, reflected by the condition number and 
maximum pairwise correlation of the design. Both measures are described in 
Section C. An OLH design has a condition number of 1 and a maximum pairwise 
correlation of 0. 


The algorithm by Ye (1998) for construction of OLH designs can be found 


in Section D. 


NOLH designs, a term coined by Cioppa (2002), relaxes the requirements 
on the orthogonal properties. He chooses the most space-filling design amongst 
a few design matrices that satisfy the near orthogonal thresholds. Typical limits 
used by Cioppa are a maximum pairwise correlation of no greater than 0.03 and 


a conditional number no greater than 1.13. 


Cioppa’s 8-step algorithm to construct NOLH designs can be found in 
Section E. 


C. DESIRABLE CHARACTERISTICS 

1. Orthogonal Measures 

Designs with good orthogonal properties are desirable because they 
provide uncorrelated estimates for the regression coefficients that analysts use to 


analyze simulation models. Two of the orthogonal measures used to examine a 


design matrix include the absolute maximum pairwise correlation, :,,,,, and the 


condition number, cond(X'X). 
a. Absolute Maximum Pairwise Correlation, 0. 
The absolute maximum pairwise correlation (Hernandez, 2006) is 


the maximum 








p,|, over all ij, where / # j, and p, is the pairwise correlation 
between columns x; and x;. The equation for correlation between any two column 
vectors ( X'and X/), where X/and X/ denotes the /" entry, while X‘and X/ are 
the means of the values in the columns, is given in Equation 2.1. 


d [i #)(x7-¥)] 


p= (2.1) 








Pamp Gives the worst correlation amongst all pairs of columns in the 
design matrix. A p,,,. Of 0 reflects an orthogonal design matrix. As a measure of 
near orthogonality for a design, a smaller value is preferred. 

b. Condition Number, cond(X' X) 

For an X, design, cond(X' X)=w1/wn, where wi and wn are the 
largest and smallest eigenvalues of X’X, when the columns are centered to 
sum to zero and scaled to the range [-1,1]. 

An orthogonal design matrix has a condition number, cond(X'X), 


equal to 1. As a measure of near orthogonality for a design, a condition number 
as close to one as possible is preferred. 


2. Space-Filling Measures 
Designs with good space-filling properties are useful for data analysis 
across the possible range of data because the experimental points are scattered 


throughout the domain in a uniform-like manner. Three measures to assess the 


space-filling properties of a design matrix are the modified L, -discrepancy (ML2), 
the centered L, -discrepancy (CL2), and the Euclidean maxi-min (Mm) distance. 
a. Modified L2-Discrepancy, ML2 
The L,-discrepancy is a measure of the non-uniformity of a design 
(Wei Chen & Sudjianto, 2003). The MLz is a variant of the L,-discrepancy and 


has been used as a space-filling measure in the construction of NOLH designs 
(Cioppa, 2002). The measure used by Cioppa is Equation 2.2, and the same 
measure in the original source (Hickernell, 1998) is Equation 2.3. 


M(x) =(2) ney IIe Xa) 
3 oe : : (2.2) 
rae > I] —max(x,;,X;;)] 
7 4 ae nek a > 
[ML,(X)] -(2) A []6- xa) 


= = ‘ (2.3) 
2 ] [[2-max(x,;,x;)] 


3 
Q 
D 
i 
XK 
i, 
T 


The two equations differ in two ways: the equation 2.3 reflects that 
MLz is a square root of Right Hand Side (R.H.S.) of the equation, and has ' in 


-k 
place of —, Equation 2.2 is used in this thesis with the values normalized to 


[0,1] in each dimension, to allow for consistency with the efforts on NOLH 
designs (Cioppa, 2002). As a space-filling measure for a design, a smaller value 
for MLz2 is preferred. 


b. Centered L2-Discrepancy, CL2 

The CLz2 is another variant for the L,-discrepancy that measures 
space-filling. The main difference from ML2, which is anchored at the origin, is 
that CL2 anchors at the center of the hypercube. The equation 2.4 (Fang, et al., 
2000) is used with the values normalized to [0,1] in each dimension. As a space- 


filling measure for a design, a smaller value for CL2 is preferred. 
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‘2 d=1 i=1 2 2 (2 4) 
pes Tae six 0.5/4 “|x 0.5| ‘x x.) 
n2 momar 2 di 2 ji : 2 di ji 
Cc. Euclidean Maxi-min Distance, Mm 


Mm is the other space-filling measure used by Cioppa (2002). For 


values in the designs scaled to the domain [-1, 1], a distance list is defined 


n 
composed of the Euclidean distance between all | pairs of design points. Mm 


is the smallest value within the list. The Euclidean distance between any two 
design points (x1, X2) is calculated in equation 2.5 (Morris & Mitchell, 1992). As a 
space-filling measure, a larger value for Mm is preferred. 





A(X,,%>) = [Ser =x0y (2.5) 


i=1 


D. ORTHOGONAL LATIN HYPERCUBE (OLH) DESIGNS 

The construction of an nxk OLH is detailed by Ye (1998). The 
dimensions of n and k can be determined from m (any positive integer). The 
number of runs, n, is given by2”+1; the number of variables, k, is given by 


2m-—2. OLH designs are denoted as O7. 


The creation of an OLH requires the construction of three matrices, M, S 
and T. 


1. Construction of Matrix M for the OLH 
The matrix M, with dimensions g xk , consists of permutations of the first 


column, a vector e representing the positive levels of a variable. Although vector 
e can be any ordering of the first g integers (1,2,....q), where qg=2”', an OLH 


design can be guaranteed when vector e is the ascending ordering of integers. 
For example, when m= 4,q=2*'=8,e=[1234567 8]’. 
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Permutation matrices, A. (created for L=1, 2,....m-—1) are used to 
generate the other columns of matrix M. Each of the permutation matrices is 
generated by the following equation, where matrix | is the 2 x 2 identity matrix, 


and matrix R= 2 ; 
1 0 


A, =/@...®1®R®...@R, where ® is the Kronecker product (2.6) 


m-1-L L 


For example, with m = 4 and L = 1, the permutation matrix A; = I@I@R (as 
seen in Table 1) yields the resultant column of Aie=[21436597]'. 
























































0 1 0 0 0 0 0 0 
1 0 0 0 0 0 0 0 
0 0 0 1 0 0 0 0 
0 0 1 0 0 0 0 0 
0 0 0 0 0 1 0 0 
0 0 0 0 1 0 0 0 
0 0 0 0 0 0 0 1 
0 0 0 0 0 0 1 0 
Table 1. A permutation matrix A, under Ye’s construction for m = 4. 


Additional permutation matrices can be created by multiplying any pair of 
A; and Aj, where i # j. In Ye’s algorithm, the columns of M are composed of 
vector e, Aje for je {1, 2, ..., m-1} and AiAmae for je {1, 2,...,m-2}. The 


m-1 
algorithm uses m-—2 of the 9 2-way combinations of the set of {A,}. 
























































0|;/0;0/0]0};]0;1/);0 
0|;/O0;0/;/0]0;,0;0/); 1 
0|;O0;0/;/0]1{,0/;]0/;0 
0|;/0;0/0]0}]1]0/;0 
0;/oO;1/0]0;0/;0/;0 
0|;/0;0/1]0};,0/;]0/; 0 
1/0]0;,0/;]0]01{0 +10 
Oo|;/1{;,0/0]0};0/;]0/;0 
Table 2. A permutation matrix A,A3 under Ye’s construction for m = 4. 





6 Given a mxn matrix A and a pxq matrix B, the Kronecker product A®B would be a 
(mxp)x(nxp) matrix C, with each element given by Cyg = ajb, where a = p(i-1)+k and B = q(j-1)+1. 
(Source: Weisstein, Eric W. "Matrix Direct Product." From MathWorld--A Wolfram Web Resource. 
http://mathworld.wolfram.com/MatrixDirectProduct.html). 
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For example, with m = 4, the first addition is the permutation matrix A1A3 
(as seen in Table 2), which yields the resultant column of A;Aze =[7 8 5 6 3 
A 27, 


An example of matrix M for m= 4, n= 17, k =6, is in Table 3. 





















































e Aye Az2e A3e A,A3e A2A3e 
1 2 4 8 7 5 
2 1 3 7 8 6 
3 4 2 6 5 7 
4 3 1 5 6 8 
5 6 8 4 3 1 
6 5 7 3 4 2 
7 8 6 2 1 3 
8 7 5 1 2 4 
Table 3. A matrix M under Ye’s construction for m = 4. 


2. Construction of Matrix S for the OLH 

Matrix S has entries +1, and has the same dimensions as matrix M, q * k. 
The first column of matrix S consists of +1’s, and the subsequent m—1 columns 
are identical to a two-level full factorial design for m—1 variables. The remaining 
m-—2 columns are created by pairwise combinations in the two-level full factorial, 
corresponding to the interactions between A, used in the creation of matrix M, 
i.e., AiA3. For example, the resultant column, i.e., S1S3, is obtained by 
multiplying, element by element, the columns of S; and $3 together. 


Ye proposed the creation of the matrix S by defining the vector aj as 


J 


—1 1 
a, =B, @B, ®...@B,,,, where B. -| and B. -| | for 


je, 2, ..., m-t}andizj. This vector replaces the columns of Sj, .., Sm1+. The 


subsequent columns are pairwise combinations of the original columns. 





1 1 
For example, with m = 4, the vector a, = Ele} \e| 4 


4 generates the 


column of S;=[-1 +1 -1 +1 -1 +1 -1 +1]". 
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An example of the matrix S for m = 4, n= 17, k= 6, is in Table 4. 


















































So S, S, S; S,S3 SS; 
+ -1 -1 -1 +1 + 
+1 + -1 -1 -1 + 
+ 1 -1 +1 -1 +1 -1 
+ +1 +1 -1 -1 -1 
+ -1 -1 +1 -1 -1 
+1 +1 -1 +1 +1 -1 
+1 -1 + +1 -1 +1 
+1 +1 + +1 +1 +1 
Table 4. A matrix S under Ye’s construction for m = 4. 





3. Construction of Matrix T for the OLH 

Matrix T, of dimensions nxk, is created by the Hadamard product’ of 
matrix M and matrix S. A row of 0’s and a mirror image of matrix T are appended 
to matrix T for the creation of an OLH. The mirror for matrix T involves changing 
the sign and flipping the order of the original matrix T. 


The resultant OLH, 0)’, withe =[1 2 3 4 5 6 7 8]', isin Table 5. 
















































































Run Var 1 Var 2 Var 3 Var 4 Var 5 Var 6 
1 1 = 74 a 7 5 
2 2 1 8 a7. -8 6 
3 3 =A 2 -6 5 57) 
4 4 3 1 -5 -6 -8 
5 5 -6 a8 4 aa oH 
6 6 5 a7 3 4 a 
7 7 8 6 2 aa 3 
8 8 7 5 1 2 4 
9 0 0 0 0 0 0 
10 26 i -5 1 a) -4 
11 7 8 -6 2 1 23 
12 -6 -5 Z 3 -4 Z 
13 -5 6 8 4 3 1 
14 a4 -3 a 5 6 8 
15 =8 4 ey) 6 -5 7 
16 =2 aA 3 7 8 -6 
17 ad 2 4 8 7 -5 
Table 5. O,’ under Ye’s construction (where m = 4, k = 6, n= 17). 


Based on the measures defined, we see that the properties for this O,’ are 


cond(X™X)= 1.0, Pym = 0.0, Mm = 1.4737, ML2 = 0.0965 and CL2 = 0.2053. 





7 Suppose A = (aj) and B = (b;) are ton x m matrices, the Hadamard product of A and B is 
the entry-wise product of A and B, that is that resultant n x m matrix has (i,j) entries of aybj 
(http://planetmath.org/encyclopedia/HadamardProduct. html). 
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E. NEARLY ORTHOGONAL LATIN HYPERCUBE (NOLH) DESIGNS 
Cioppa describes the construction of nxk OLH and NOLH designs by 
extending Ye’s algorithm (Cioppa, 2002), increasing k from 2m-2 _ to 


m-1 
m( 5 } with the number of runs, n, remaining constant. NOLH designs are 


denoted as N,’. 


The matrix M is extended by using all possible two-way combinations and 
appending the additional columns, AjAje for /,/=1, 2, ....m-1andi+j. For 
example, for the case where m = 4, the extension appends the columns, {A1Aze, 
A,A3e, A2A3e} instead of only {A;Aze, AzA3e}. This increases the number of 
variables explored, as depicted in Table 6. Construction of matrix S then 
requires the same number of additional columns, while the construction for matrix 


T remains unchanged. 






































m n k (Ye) k (Cioppa) 
4 17 6 7 
5 33 8 11 
6 65 10 16 
7 129 12 22 
8 257 14 29 
Table 6. Comparison for k between Ye and Cioppa. 


By varying the order of e in the creation of the matrix M, different design 
matrices, which yield different properties, can be constructed. The original vector 
e=[1234567 8 provides an orthogonal design, but tends to have poor 
space-filling properties. There are a total of (2”")! permutations to vector e. 
This creates problems of searching for the best, as seen in the large numbers in 
Table 7. 


























m Number of Permutations 

4 40320 

5 20922789888000 

6 263130836933694000000000000000000000 
Table 7. Number of possible permutations of vector e. 
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The 8-step construction proposed by Cioppa is detailed (Cioppa & Lucas, 
2006) as follows: 
Step 1: Determine the number of variables (k > 7) required. If the number of 


m-1 
variables is other than 11, 16, 22, or, more generally, ims 3 p round the 
required number of variables up to the nearest one of these numbers. 


Step 2: Establish a maximum threshold pairwise correlation value and a 
maximum threshold condition number. Based on extensive experimentation, we 
use ,,,, = 0.05, 0.17, 0.16, and cond(X' X) = 1.15, 2.4, 2.8 for k = 11, 16, 


and 22, respectively. 
Step 3: Using a randomly permuted e, construct a design matrix. 


Step 4: Calculate the pairwise correlations and the condition number of the 
candidate matrix. 


Step 5: If either value in Step 4 exceeds the thresholds in Step 2, discard the 
design and return to Step 3 to regenerate with another randomly permuted e. 
Otherwise, keep the design and proceed to Step 6. Repeat Steps 3-5 until a 
desired number of candidate designs are found. Note: If not enough are found, 
relax the criteria in Step 2 and begin again. We have found that 15 candidate 
designs work well for Steps 6-8. 


Step 6: Subject each of the candidate designs to repeated applications of 
Florian’s method to decrease the maximum pairwise correlation and condition 
number. Stop when no further improvement is made. 


Step 7: Calculate the Mm distance and ML2 discrepancy for each of the Step 6 
designs. Rank the designs according to some combination of these measures. 
We chose the design with the minimum rank sum over the two measures. 


m-1 
Step 8: If a number of variables other than 7, 11, 16, 22, or m “| 3 ) is 
required, construct each possible subset having the appropriate number of 
columns from the Step 7 design and calculate the Mm distance and ML2 


discrepancy. Choose the design with the best combination of these two 
measures. 


With preset thresholds of cond(X'X)<1.13 and p,,, <0.03, Cioppa 


identifies 15 designs that satisfy the definition of near orthogonality. The one 
with the best space-filling property is chosen (as in Step 7) and catalogued as the 


best NOLH design, N,’, for the particular n and k values. 
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F. EXTENDING OLH AND NOLH DESIGNS 

With Cioppa extending Ye’s algorithm to include all 2-way combinations 
and still maintaining orthogonal properties, the exploration into the inclusion of 3- 
way and larger combinations for the permutation matrices for construction of the 
LH designs was performed. Through the analysis, it is confirmed that the 
orthogonal properties are maintained, thus allowing the ability to investigate a 
larger number of factors with the same number of design points in an OLH 


design. 


For each design matrix, the inclusion of p-way combinations also includes 
all smaller combinations for the permutation matrices. For example, an inclusion 
of a 4-way combination also includes both the 2-way and 3-way combinations. 
The number of factors, k, with the inclusion of p-way combinations (where the 


maximum is p< m-—1), is given by the following equation: 


k -15(7"") (2.7) 


The number of factors for each design is improved as shown in Table 8. 



























































e =|) VES Cioppa Ang 
OLH (2-way) 3-way 4-way p-way8 

9 3 4 4 - : 4 

17 4 6 7 8 - 8 
33 5 8 11 15 16 16 
65 6 10 16 26 31 32 
129 7 12 22 42 57 64 
257 8 14 29 64 99 128 
513 9 16 37 93 163 256 
1025 | 10 18 46 130 256 512 

Table 8. Comparison of LH designs with p-way combinations. 


The construction of the LH designs remains similar to the methods 
proposed by Ye, and also by Cioppa. The additional permutation matrices 


include all the combinations of A,A,...A, for xX,,X,,..X,¢12,....m-1, and 


X, #X_ #...#X,. For example, for p = 3, the 2-way combinations AjAj, for i # /, 


8 The maximum possible p-way combination depends on the value of m, where p=m-1. 
For example, if m = 4, the maximum p-way combination is a 3-way combination. 
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and 3-way combination AjAjAx are included for the permutation matrices, for all 
i,j,k e1,2,....m-1 and iz jk. 


For m = 4, the extension appends the column, {A,A2As3e} in addition to the 
2-way combinations. Construction of matrix S includes the additional number of 
columns in a similar fashion. The construction of matrix T remains unchanged. 


The details for the construction of the design matrix O;’ can be seen in Appendix 


A. 


With the inclusion of additional combinations for more factors, there are 
tradeoffs in the orthogonal and space-filling properties for the individual designs. 


For orthogonal properties, the extended O;’ has the same properties as the 


methods by Ye and Cioppa. The space-filling properties for the designs with 
larger numbers of combinations will be worse when compared with the other 
designs, because we are putting the same number of design points in a larger 
space. A table of comparison is shown in Table 9. 














cond(X™X) ane Mm ML2 CLo 
Ye 0, 1.0 0.0 1.4737 | 0.0965 | 0.2053 
Cioppa O}’ 1.0 0.0 1.479 | 0.1732 | 0.2551 
Ang O,’ 1.0 0.0 1.7854 | 0.3144 | 0.3142 


























Table 9. Properties of O}’, O7’ and O}’. 


For designs with a fixed number of runs, n, there are usually tradeoffs for 
increasing the number of variables, k, for the design. The increase in k, 
increases the dimensionality of the design matrix, and improves Mm. However, 
the ML2 and CL2 measures reflect worse space-filling properties for a larger k. 


For NOLH designs, the limits used are cond(X' X) <1.2 and Pamp < 9.05. 
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lll. DISCRETE/CATEGORICAL VARIABLES AND THEIR 
EFFECT ON PROPERTIES OF LH DESIGNS 


A. INTRODUCTION 

A discrete variable is a variable that assumes values within a discrete set, 
such as integers. Discrete variables can either be nominal or ordinal variables. 
Nominal variables are variables where order does not matter, such as gender. 
Ordinal discrete variables have an ordering or quantitative value associated, e.g., 


the number of tanks and military ranks. 


An example of a nominal discrete variable is gender. Male and Female 
are often represented in designs in the continuous range by values 1 and 0, or 0 
and 1, respectively, as there is no correlation between the representation and the 
nominal variable. Another example of a nominal discrete variable with more than 
two levels is the military service, which include Army, Navy, and Air Force. In 
general, nominal discrete variables with a levels require a@-—1_ indicator 


variables. 


An example of an ordinal discrete variable is the number of soldiers or 
tanks, which can take only integer values. Another example of an ordinal 
discrete variable is the military ranks for the Singapore Army, which have an 
ordinal scale. 


Our investigation of the effects of discrete variables on the properties of 
the LH designs seeks to answer the following questions: 


e How does the choice of the column used for the discrete variable 
affect the properties of the LH design? 


e What are the effects on the properties of the LH design for 


increasing numbers of discrete variables? 


e How do different levels for discrete variables affect the properties of 
the LH design? 
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e How does different coding of discrete variables affect the properties 
of the LH design? 


B. METHODOLOGY OF EVALUATION 

The methodology used to investigate the effects of discrete variables on 
the properties of LH designs looks at the effects of representing, coding and 
ordering of discrete variables. OLH and NOLH design matrices constructed 
earlier are used, and the results of this evaluation are discussed in Chapter IV. 

1. Representing Types of Discrete Variables 

Both nominal and ordinal discrete variables are investigated. Ordinal 
discrete variables are more significant, as the presence of ordinal variables is 
extremely common in real world problems. This thesis looks at the coding 
methods for both nominal and ordinal discrete variables in the following sections. 

2. Coding Discrete Variables 

Coding of nominal discrete variables with {high, low} can be done in (at 
least) two different methods, because the ordering of the values does not matter. 
The two methods we look at are positive-negative coding and odd-even coding. 
Positive-negative coding assigns all positive values in our earlier LH designs with 
the high value and negative values with the low value. Odd-even coding assigns 


all even numbers with the high value and the odd numbers with the low value. 


For example, both coding for the O;’ design are shown in Table 10. 











Original: |) })-85) 279 |-6s |e eA 310-2) 4 | Ot | 2) Bae | | ee | 7) 8 
Positive- 

lo | lo | lo | Lo} lo} lo | lo | lo | hi | hi | hi | hi | hii | hi | hi | hi | hi 
Negative 
Odd- 

hi | lo | hij Lo} hij lo | hi} lo | hij lo | hij lo | hij lo | hij lo | hi 
Even 






























































Table 10. Coding methods for nominal variables. 
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The mathematical formulation of the positive-negative (PN) and odd-even 
OD) coding are: 


max value if x>0 


COdING py, = 
nee ies value if x <0 


remainder = MOD\value, 2) 
max value if remainder =0 


codingoe =4 
min value _ if remainder #0 


Coding discrete variables with more than 2 levels is done by distributing 
the levels uniformly across the number of levels of the discrete variables. For 
example, the coding for the 3-level ordinal discrete variable {0, 1, 2} in the 


O;’ design is represented in Table 11. 





Original | -8 | -7 | 6 | -5 | -4])-3)-2/-1}/0]1],2)/3)4/5/6|7) 8 





Coding |} 0/0;}0/)/0/0]0;1;,1;);1/1/,1]2)2),2]2)2 42 
Table 11. Coding for ordinal variables. 






























































3. Ordering Columns of a LH Design for Additional Discrete 
Variables 


For a single discrete variable, the best column to represent the discrete 
variable is obtained after comparing the orthogonal and space-filling properties of 
the design with the discrete variable coded. All of the columns are used for the 
single discrete variable to generate additional designs. The properties for each 
new design are ranked, and the column which gives the best properties should 
be used to represent the single discrete variable. 


For example, the O}’ design is used to represent a single 2-level discrete 


variable. After generating 7 new designs, the properties of the new designs are 
collected and ranked, as seen in Table 12. The best of the new designs (with the 


lowest overall rank) is the choice of the column for the 2-level discrete variable. 


Column 5 is the best choice for O;’ to represent the 2-level discrete variable. 
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Raw results Ranking 


























Col 
Used | 2% | Pam | Mm | MLz | conve | Pare | Mm | ML, | Sverall 
1 4.004 0.385 1.431 0.627 4 2 3 1 10 
2 4.004 0.385 1.392 0.629 4 2 4 2 12 
3 4.004 0.385 1.458 0.640 4 2 2 4 12 
4 4.004 0.385 1.275 | 0.665 4 2 6 7 19 
5 | 2946 0.193 | 1.474 | 0631 | 1 1 1 3 | 5 (Best) 
6 | 3710 0.385 | 1.341 | 0.648) 2 2 5 6 15 
7 | 3928 0.385 | 1.192 | 0.645] 3 2 7 5 17 






































Table 12. Ranking and selection of columns for discrete variables. 


The selection of columns for new LH designs for multiple discrete 
variables will choose the best column, and sequentially choose the other 
columns that give the best orthogonal and space-filling properties. An iterative 
process determines the columns, and although this process may not provide the 
best (or global optimal) result, the saving of computational time promotes its use 


over the total enumeration of all possibilities. 

The ordering of columns to use for the designs, for additional 2-level 
variable columns for the O)’design, is {5, 2, 6, 3, 7, 4, 1}. An example of 
interpreting this result for a scenario with two 2-level discrete variables using the 
O}’ design is that we should use columns 5 and 2, because they provide the best 


orthogonal and space-filling properties. This is further described with illustration 
in Chapter IV, Tables 19 and 20. 
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IV. RESULTS OF RESEARCH 


A. SOFTWARE IMPLEMENTATION OF LATIN HYPERCUBE 
GENERATOR IN JAVA 


Within the IDE, running primarily on a personal laptop’® or lab 
computer!1, OLH and NOLH designs corresponding to Table 13 were 
successfully generated. Properties of the individual designs are discussed in 


section B. The largest OLH design, O;°°, usinge =[1 2 ... 512 ]', took 


approximately 30 minutes to generate. The time taken to generate different 
NOLH designs is largely dependent on the number of candidate designs 





























considered. 
= = Additional p-way combinations for the permutation matrix 
2-way | 3-way | 4-way | 5-way | 6-way | 7-way | 8-way | 9-way 
17 4 7 8 NA NA NA NA NA NA 
33 5 11 15 16 NA NA NA NA NA 
65 6 16 26 31 32 NA NA NA NA 
129 7 22 42 o7 63 64 NA NA NA 
257 8 29 64 99 120 127 128 NA NA 
513 | 9 ar 93 163 219 247 255 256 NA 
1025 | 10 46 130 256 382 466 502 511 512 






































Table 13. List of LH designs generated. 


Inclusion of p-way combinations of the permutation matrices during 
construction of OLH and NOLH designs can achieve a larger LH design. The 
catalogue of OLH designs captures all designs, and the catalogue of NOLH 
designs is still in progress. The catalogues can be found on the SEED Center 
website (http://harvest.nps.edu); some of the designs are catalogued in Appendix 
B. 


The code for the generation of OLH and NOLH is also available on the 
SEED Center website. 





9 Eclipse SDK (Version: 3.2.0) Build id: M20060629-1905 (Available via: 
http://www.eclipse.org/downloads/). Last date accessed, Nov 2006 


10 Home laptop — Fujitsu running Intel® Pentium® M 1.73 GHz, 1.00GB RAM. 
11 Lab computer — Dell running Intel® Pentium® 4 3.40 GHz, 2.00 GB RAM. 
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Construction of NOLH designs are done in two ways, with varying 
performances in speed and properties of LH designs, namely random and 
sequential. The random permutation of the vector e, generates NOLH design in 
slightly longer time, but with good properties. The sequential method searches 
around a known OLH design and generates the NOLH design much faster, but 
its properties are dependent on the initial starting point. An illustration of the idea 


for both methods is shown in Figure 3. 


@ Candidate Design from 
random permutated e vector 


@ Candidate Design from fixed 
permutated e vector about a 
known OLH 


@ Known OLH 





Figure 3. Illustration of the random and fixed permutation of the e vector for 
construction of the NOLH designs (Best viewed in color). 


4; Summary of Functions within JAVA Application 
The JAVA application is able to do the following: 


e Construct OLH to O} to Oj{2°. 
e Construct NOLH fromN;to N33? (work is still ongoing as 2'° “is 
forever”). 


e Read in LH design (in .txt format) and output the orthogonal and 


space-filling properties of the design. 


24 


e Read in LH design (in .txt format) and perform a categorical 
analysis on the design matrix, and output the order of the columns to 


use which give good properties to the now, mixed design. 


e Read in LH design (in .txt format) and stack the design based on 
user’s input, and output the design and its corresponding orthogonal 


and space-filling properties. 


2. Catalogues of OLH and NOLH Designs 
A partial catalogue of OLH designs is included in Appendix B. Current 


designs include the construction for m<10 (up to O92) and are available on the 


SEED Center website. 


A catalogue of NOLH designs, where the thresholds are cond(X' X)<1.2 
ANd Pam, <0.05, is available on the SEED Center website. Construction of 


NOLH designs will continue to obtain better design matrices, and updated design 


matrices will also be available. 


With the ability to extend the OLH and NOLH designs by the inclusion of 
additional p-way combinations of permutation matrices, the design matrix of Of is 


in Table 14. 


25 



































RIS] EO] e|Nl@}a]s}o}r}ja 





12 



























































-30 





-29 



















































































-20 








-21 





-22 





-23 








-24 


foe) 





-25 


-9 





-26 


-10 





-27 


-11 





-28 


-12 





-29 


-13 





-30 


26 


-14 





-31 


-27 


-15 








-32 


























-18 





-2 





-28 

















-16 





-27 





-19 





-3 





-23 






































Table 14. 


Design matrix of Of. 
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B. PROPERTIES OF OLH AND NOLH DESIGNS 
1. Experimenting and Extending the OLH and NOLH Designs 
Through experimentation, new design matrices to include up to p-way 
combinations of permutation matrices were constructed. These designs have 


maintained their orthogonal properties of cond(X'X)=1.0 and Pamp = 9-0, even 
with the inclusion of 3-way, 4-way, ..., (m™-—1)-way combinations when the vector 


eis [1, 2, ..., 2™']'. The properties for the LH designs that include 3-way and 4- 
way combinations are in Table 15, in comparison with the mean of 100 Flexible 
Random Latin Hypercube (FRLHS) designs. 



































3-way 4-way Mean of 100 FRLHS 
mion k cona(x'x) | Mm k cona(x™x) | Mm k cona(x’x) | Mm 
Pamp ML2 Pamp ML2 Pamp ML2 
1.0 1.78 2.47 1.27 
4) 17 | 8 00 | 034 | 8 MS Me 8 0.24 | 0.35 
1.0 2.20 1.0 2.42 2.11 1.96 
9 33 15 0.0 7.01 16 0.0 11.52 15 0.17 6.23 
1.0 2.10 1.0 3.18 1.77 2.17 
6 65 26 0.0 8.67E2 31 0.0 1.04E4 26 0.10 4.31E2 
1.0 3.27 1.0 4.14 1.54 3.27 
7 | 129 42 0.0 1.76E6 o7 0.0 5.24E9 42 0.07 1.78E5 























Table 15. Construction of OLH designs with 3-way and 4-way combinations. 


Orthogonal properties of the LH designs are maintained with the inclusion 
of higher order combinations, but these designs have less ideal space-filling 
properties. Depending on the type of scenario and the limitations to the number 


of runs, different design matrices may be used. 


After the initial test with additional columns confirmed success _ in 
generating OLH designs, additional OLH designs were generated and the 


properties of the designs are shown in Table 16. 
Observations for these designs include: 


a) Increasing the number of possible combinations of the permutation 
matrices increases the number of factors for the design matrices. 
However, the space-filling nature is less favorable in comparison with 


designs with fewer combinations of the permutation matrices. 
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k n cond(X™ X) Ones Mm ML, CL; 
7 17 1.00 0.00 1.48 0.17 0.26 
8 17 1.00 0.00 1.79 0.31 0.31 
11 33 1.00 0.00 1.67 0.95 0.41 
15 33 1.00 0.00 2.20 7.01 0.73 
16 33 1.00 0.00 2.42 11.52 0.83 
16 65 1.00 0.00 1.79 7.98 0.72 
22 | 129 1.00 0.00 1.79 96.58 1.39 
26 65 1.00 0.00 2.70 867.17 2.25 
29 | 257 1.00 0.00 1.81 1701.28 2.92 
31 65 1.00 0.00 3.19 1.E+04 3.816 
32 65 1.00 0.00 3.34 2.E+04 4.261 
37 | 513 1.00 0.00 1.86 5.E+04 6.81 
42 | 129 1.00 0.00 3.27 2.E+06 11.34 
46 | 1025 1.00 0.00 1.91 2.E+06 18.24 
57 | 129 1.00 0.00 4.14 5.E+09 49.27 





Table 16. Properties of OLH designs (from O7’ to O/?°). 


b) The results shown are intuitive. For example, comparing the 
designs O*° and O72, shows that a design with more runs, n, improves the 
space-filling property of the design for a fixed number of variables. 

Cc) The orthogonal properties of an OLH design (where cona(x™x) = 1.0, 
Pam = 9-0) are dependent on the permutation of vector e. With a total 
enumeration of the possible permutations of vector e for a 17 run design, 
there are a total of 640 O}’and 640 O;’. These designs result from the 


same vector e that is used in the construction. The total enumeration of 
larger designs is not possible because the total number of permutations of 


the vector e becomes too big, as shown earlier in Table 7. 


Comparing these 640 designs, the properties for the best space- 
filling designs are in Table 17. The best designs show improvements to 


the space-filling property, ML2. A pairwise plot of the best O’ design is 


shown in Figure 4. 
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k n cond(X™ X) Oa Mm ML, CL, Remarks 

7 17 1.00 0.00 1.48 0.17 0.26 e = [1,2,..,8] 
7 17 1.00 0.00 1.479 0.1519 0.2426 Best of 640 
8 17 1.00 0.00 1.79 0.31 0.31 e = [1,2,..,8] 
8 17 1.00 0.00 1.7854 0.2893 0.3003 Best of 640 





Table 17. Properties of O)’ and ©;’ with total enumeration of e. 


os 
fad ce 
baa 


Column 5 


ba 


Pe al 





Figure 4. Pairwise plot of the O;’ design. 
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d) The extension of Ye’s algorithm maintains the following two 
theorems (Ye, 1998): 


Theorem 4 (Ye, 1998). For the LH design constructed, 
the elementwise square of each column is orthogonal to all the 
columns in the LH design. 


Theorems 5 (Ye, 1998). For the LH designs constructed, 
the elementwise product of every two columns is orthogonal to 
all columns in the LH design. 


The result of the two theorems provides subsequent estimates of 
quadratic and bilinear interaction effects that are uncorrelated with the 
estimates of the linear effects. 


2. Exploring the Construction of Latin Hypercube Designs 

OLH and NOLH designs can be constructed by different methods, and two 
methods of construction were compared against catalogued designs. The two 
methods include random generation and using algorithms proposed by Ye and 
Cioppa. The driving factors for this investigation are the speed and ease of 
construction vis-a-vis the orthogonal and space-filling properties of the LH 
designs. 


From the 100 designs generated from Random Latin Hypercube Sampling 
(RLHS)12, Florian’s method was performed on the RLHS designs, obtaining new 
designs called Flexible RLHS (FRLHS) designs with improvements to the 
orthogonal properties. The mean of these 100 designs are used for the 


comparison. 


Designs that were constructed from the algorithm include OLH and NOLH 


designs and the existing catalogue of designs from the SEED Center website. 





12 RLHS method used MATLAB’s X = Ihsdesign(n,p) that generates a Latin Hypercube 
sample X containing n values on each of p variables. The RLHS refers to the average of 100 
RLHS designs. FRLHS are Flexible RLHS design that used Florian’s method to improve the 
orthogonal properties. 
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A subset of the comparison of the properties can be seen in Table 18, with 


a larger compilation of results seen in Appendix C. 


















































K N Type cond(X" X) Pan Mm ML2 
OLH 1.00 0.00 1.48 0.17 

SEED 1.00 0.00 1.48 0.15 

7 17 NOLH 1.17 0.05 1.44 0.16 
RLHS 9.53 0.52 0.97 0.57 

FRLHS 2.24 0.23 1.11 0.19 

OLH 1.00 0.00 1.79 0.31 

SEED NA NA NA NA 

8 17 NOLH 1.06 0.03 1.79 0.30 
RLHS 14.41 0.57 1.12 0.92 

FRLHS 2.47 0.24 1.27 0.35 














Table 18. Properties of LH designs of different construction (for n = 17). 


Four properties were considered while comparing the different LH 
designs: they are the orthogonal properties cond(X"X) and p,,,,; and the space- 
filling properties Mm and ML3. 

The comparison of the properties is based on the results for 50 different 
designs, using 11 designs (each) from the RLHS, FRLHS, OLH and NOLH 
designs; and 6 designs from the NOLHdesign_v4.xls (available on the SEED 


Center website). 


In comparing the speed and the ease of construction, the RLHS designs 
and the NOLH designs from SEED are the easiest and fastest, followed by the 
FRLHS designs, OLH designs, and NOLH designs. 


A parallel plot of the properties for the different LH designs based on 


different construction methods can be seen in Figure 5. 
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BETTER PERFORMANCE 











cond(X XxX) Pammp Mm ML2 CL2 


FRLHS (11) Umea OLH (11) NOLH (18) 
Yj \ Jv 
a WF < 


FAST CONSTRUCTION —> SLOW CONSTRUCTION 


Figure 5. Comparison of speed and ease of construction vs. properties of LH 
designs (Best viewed in color). 























The parallel plot in Figure 5 reflects the improvement of Florian’s method 
on the RLHS to RLHS-F designs, as seen in charts (b) and (c). The odd line of 
the OLH designs (chart (d)) is due to poor space filling properties of larger 
designs. OLH and NOLH designs show similar space filling properties, 
comparable to the average of 100 RLHS and RLHS-F designs. The notable 
differences between the different construction methods for the LH designs are 
that: 


a. Florian’s method improves the orthogonal properties of the design 
matrices of the RLHS, and the improvement is largely dependent on the n 
and k values. The comparison of matrices with similar n and k reveals that 
the orthogonal properties are slightly worse than the OLH and NOLH 
designs. However, RLHS designs have attractive space-filling properties. 
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b. For smaller design matrices, the space-filling properties do not 
differ greatly from the OLH with the NOLH or RLHS. However, the Mm 
and ML2 measures tend to reflect a downgraded performance with larger 


design matrices. 


C. The comparison between design matrices with the same number of 
runs, n, i.e., including the design matrices with 2-way, 3-way and 4-way 
combinations of permutation matrices, reveals that the inclusion of higher 
order combinations of permutation matrices causes downgraded 


properties for cond(X™X); Pamp» and ML2, but an improved Mm property, 


as there is a bigger space in which to push the design points apart. 


3. Exploring the OLH and NOLH Designs for Discrete Variables 

Exploration into the representation of discrete variables, both nominal and 
ordinal, is important to fitting a scenario into a chosen design matrix. From a 
user’s point of view of a NOLH tool for the design of experiments, having in-depth 
knowledge on the method of construction for the different designs for different 
scenarios may not be important. But, given a choice, the user should prefer a 
better design if available. 


The exploration looked into the modifications of an LH design with the 


inclusion of discrete data, to find the “best” design from the original LH design. 


Interesting results were obtained from the study into the properties of OLH 
and NOLH designs with different numbers and levels of discrete variables. 


a. For 2-level discrete variables, Odd-Even coding leads to problems 
with increased numbers of variables. Although the coding provides for a 
better orthogonal property with the addition of a single discrete variable 
with 2-levels, it leads to problems when additional 2-level discrete 
variables are to be included for the design. The comparison between 
successive additions of 2-level discrete variables on O)’with different 


coding methods is shown on Table 19. 
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For example, when the O/’ with Odd-Even coding requires three 2-level 


discrete variables, it has p,,,=1. The result of p,,,=1 is also reflected for all 


other OLH and NOLH designs. 


Properties of O7-17 with Pos-Neg Codin Properties of O7-17 with Odd-Even Codin 

cond(X™X) Pamp Mm ML, cond(X7X) Pamp Mm ML, 
1.00 0.00 1.48 0.17 igi 1.00 0.00 1.48 0.17 
2.946 0.192 1.474 0.631 2.667 0.000 1.458 0.622 
3.817 0.385 1.244 1.062 16.000 0.889 1.639 1.165 
3.042 0.096 1.257 1.483 1.6E+17 | 1.000 1.620 1.868 
4.355 0.385 1.125 1.912 3.4E+17 | 1.000 0.612 2.492 
3.613 0.192 1.008 2.349 1.5E+49 1.000 0.559 3.386 
4.743 0.385 1.000 2.840 Infinity | 1.000 0.250 4.028 
1.438 0.056 0.000 3.281 Infinity | 1.000 0.000 5.070 

Table 19. Problems with Odd-Even coding with addition 2-level discrete 

variables (demonstrated on O}’). 
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Order of Columns 





























The O}’design, with Positive-Negative coding, starts with cond(x'™x)=1 
and ~., =0 (as seen in Table 19). If one 2-level discrete variable is required, the 


use of column 5 gives the best properties, as reflected in the chart. If two 2-level 
discrete variables are required, the use of column 5 and column 2 results in a LH 


design with better orthogonal and space-filling properties. 


cond(X' X) cond(X' X) 
1.00 igi 1.00 
2.946 2.805 
3.817 3.613 
3.042 2.567 
4.355 3.926 
3.613 2.805 
4.743 3.845 
1.438 1.000 

Table 20. | Pos-Neg coding with additional 2-level variables for O7’ and O°. 
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Removing the centre row of zeros for O)’ obtains the O/° design. The 


effect of additional 2-level discrete variables results in an orthogonal design when 
all the columns are used to represent 2-level discrete variables (as seen in Table 
20). This result holds true for the OLH designs. 
b. For ordinal variables, the number of levels, /, for the discrete 
variables affects the performance of the design matrix. Each column will 
only have values from 1,2, .., / if the column is used to represent an /-level 
discrete variable. For example, if representing a 3-level discrete variable, 
the column used in Of” willbe [12312312312312312]'. 


The properties (in particular, the orthogonal measures) of O}’ are affected 
by fewer levels, and the results stabilize as the levels increase. Figure 6 shows 
the properties for O7’ with different levels of discrete variables, similar for other 


OLH and NOLH designs. 
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Figure 6. Comparison of properties when one /-level discrete variable is added to 
the O)’ design (Best viewed in color). 
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Cc. There is no effect on the orthogonal properties for the addition of 


discrete variable columns for “complete”'3 designs, which includes O,;’, 
33 65 1025 “ ” . + gs . 
Oj; , O32, ---, Ogio. “Complete” designs do not have variations in the 


orthogonal properties when the different columns are selected, unlike the 
smaller designs which result in variation in the orthogonal properties. 
However, some of the space-filling properties have minor changes. 
Though with poorer space-filling properties compared to “not complete” 
designs, this property could be useful to note while considering the design 
to use, i.e., the use of the OLH and NOLH designs with a larger number of 


discrete variables. 


4. Exploring the Effects of Stacking on OLH and NOLH Designs 

Stacking with random permutation of the columns improves the space- 
filling properties of LH designs. The basic principle behind stacking is that the 
random permutation of columns maintains the orthogonal properties of an OLH 
design. Adding additional copies of the designs with random permutation 
improves the space-filling properties while maintaining orthogonal properties. 


Furthermore, these are easily constructed and allow replication at each level. 
The notation of s:O; describes a design with an O, appended with s—1 
other O,; with randomly permuted columns. For example, the illustration in 


Figure 7 shows the construction of 2:05. 




















































































































1 -2 1 -2 -2 1 1 -2 
2 1 2 1 1 2 2 1 
0 0 -1 2 -1 0 0 
-1 2 -2 -1 -1 -2 -1 2 
-2 -1 -2 -1 
-2 1 
Random 1 2 
permutation of 2 -1 
= é O; design columns of O} A 2 
Original O; (with row of Os design (with row 5. 
design removed) of Os removed) 2:05 design 
Figure 7. Stacking of O} to form a 2:03 design. 


13 “Complete” designs refer to designs where the construction includes all possible p-way 
combinations of the permutation matrices. 
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designs are captured in the Table 20. 


The orthogonal and space-filling properties of the best of five s:O; 





















































Ox Number of Stacks (s) 
Original 
n k Properties Matrix (1) 2 5 10 15 
Mm 1.369 0.500 0.354 0.000 0.000 
9 4 ML2 0.054 0.053 0.054 0.057 0.058 
CL2 0.184 0.179 0.173 0.173 0.174 
Mm 1.785 1.212 0.829 0.530 0.559 
17 8 ML2 0.314 0.203 0.156 0.129 0.128 
CL2 0.314 0.245 0.196 0.171 0.167 
Mm 2.417 1.620 1.395 1.038 1.104 
33 16 ML2 11.525 6.324 3.211 2.306 1.828 
CL2 0.834 0.605 0.403 0.313 0.262 
Mm 3.342 2.036 1.952 1.680 1.809 
65 32 ML2 1.72E+04 | 1.03E+04 | 5.22E+03 | 3.51E+03 | 2.98E+03 
CL2 4.261 3.043 1.943 1.385 1.135 
Mm 4.673 3.225 3.055 2.660 2.772 
129 | 64 ML2 3.01E+11 | 1.06E+11 | 4.87E+10 | 2.96E+10 | 2.28E+10 
CL2 98.993 70.275 44.563 31.554 25.075 
Table 21. Properties of s:0, for different numbers of stacks. 


The points of interest on the effect of stacking to improve the space-filling 


properties of LH designs include: 


a. Random permutation of columns being stacked provides 
improvements to the space-filling properties. But it does not provide the 
design with the best properties. Total enumeration is extremely tedious as 


the total possible permutations is given mathematically as 


Total permutations = (k!)*s. (4.1) 


b. For the smaller designs (such as O} and O?), the Mm property 


becomes 0 due to the randomized permutation repeating the design points 
due to the stacking of additional matrices. The other space-filling 
properties of ML2 and CL2 show worse performance with an additional 


number of stacks, possibly due to the same reasons. 
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C. For the larger designs, although the Mm measure becomes worse 
for all cases, this is due to additional design points being added to the 
design matrix. The other space-filling properties of ML2 and CL2 show 


improvement in performance with the increased number of stacks. 


38 


V. CONCLUSIONS AND FUTURE RESEARCH 


A. CONCLUSION OF RESULTS 

From the OLH designs proposed by Ye to the NOLH designs by Cioppa, 
the need for bigger and better designs can be seen. Bigger designs allow for the 
design of larger computer simulations and experimentation. This, in turn, 
enables more comprehensive analysis. The inclusion of additional combinations 


during construction, and the use of JAVA for construction, extends the current 


catalogues of OLH and NOLH designs to include Of/5° and N;°3° designs. 


The presence of discrete variables is common in most computer 
simulation and experiments. This thesis provides a useful guide for knowing 
which columns to change in order to maintain the best orthogonal and space- 
filling properties; how the effect of different levels affects the properties of the LH 
design; and, how the use of positive-negative coding is best suited for 2-level 


nominal variables. 


The space-filling properties of LH designs are improved by stacking the 
designs. The LH generator in JAVA allows for the creation of stacked designs as 
desired. 


B. FUTURE RESEARCH 

A subsequent literature review led to a proposed methodology (Steinberg 
& Lin, 2006) for creating an additional increase in the number of factors for an 
OLH design. However, the construction method could not be found to access 
other matrices with similar constructs to compare the space-filling and orthogonal 
properties. An investigation into this method of construction for the LH designs 
would be an interesting study, to determine the space-filling properties of the 
designs in comparison with designs by Ye and Cioppa. 


Further analysis of the predictive powers of the extended LH design 
(which includes both OLH and NOLH designs) is suggested. NOLH designs 
have orthogonal properties thresholds for selecting candidate designs. It would 
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be interesting to discover how these limits affect the predictive powers of the 


designs. 


Within the milieu of the JAVA application to generate and access the 
properties of the LH designs, a smarter and more efficient method to search the 


space could be investigated to improve the speed of generating the design. 
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APPENDIX A. EXTENDING THE ORTHOGONAL LATIN 
HYPERCUBE FOR M = 4 


1. Construction of M for the OLH 0,’ 


The e vector usedise= [1234567 8]. 


ForL=1,Ale=[2 1 4 3 6 5 8 7]' where A1 = /@/@R yields 
the following permutation matrix. 
0 0; 0/0 























OlO;O;/O/;/O/;O/}— 


























-/O;O;}O;/O/O/O/}O 
-/O;/O;O;O/O/O 








OJ/O/O;/O/;/O/;O};— 
OJ/O/O;/O/}/-]/O;O 
OJ/O/O;/O/;/O;-|O 


0 
ForL = 2, A2e=[4 3 2 1 


0 
5]' where A2= /@ROR. 





0 
0 | 0 
0 | 0 
0 | 0 
O | 1 
1 | 0 
0 | 0 
0 | 0 
876 


ForL=3,A3e=[8 7 6 5 4 3 2 1]T whereA3= R@ROR. 


Additional permutation matrices are constructed, for example A1A3 
that would obtain A1A3e =[7 8 5 6 3 4 1 2]T, yields the following 


permutation matrix. 
























































OO OO Oe Oe) A 
Oe Oot Oe i Oe Sy) Oh Oil ef 
0;0;O0;O0/;/1;0|0|0 
OF 0. Oe ON Oe eT ai Oe ji 
Oe Oe Oe 0 Oe) Oe 
OO Oo hel) Ore O i OF pO 
hand OR Oe Ose Oe Oe OS | 0 
Oot Oe OO. Oe) Oe 0 
A1A2, gives A1A2e=[3 4127 85 6]' 


A2A3, gives AZA3e =[5 678123 4]' 


A1A2A3, gives A1A2A3e =[6 5 8 7 21 4 3]' 


41 


Thus M would be the following matrix. 


















































e Ay A2 A3 A,A2 Ay A3 A2A3 Ay A2A3 
1 2 4 8 3 7 5 6 
2 1 3 7 4 8 6 5 
3 4 2 6 1 5 7 8 
4 3 1 5 2 6 8 7 
5 6 8 4 7 3 1 2 
6 5 7 3 8 4 2 1 
7 8 6 2 5 1 3 4 
8 7 5 1 6 2 4 3 








2. 


From the original matrix with So, Si, ... 


Construction of S for the OLH 0,’ 


, 93, additional columns are 


generated. These additional columns are just multiplications of the two columns 
together, and include S1S2, S1S3, S253 and S1S2S3. 
































So S; S2 S3 S1S2 S183 S2S3__| $1S2S3 
+1 FT 1 1 +1 +1 +1 1 
+1 +1 1 1 1 1 +1 +1 
+1 1 +1 1 1 +1 “1 +1 
+1 +1 +1 1 +1 FT yi 1 
+1 1 1 +1 +1 FT 1 +1 
+1 +1 1 +1 yl +1 1 1 
+1 1 +1 +1 1 1 +1 1 
+1 +1 +1 +1 +1 +1 +1 +1 


























3. 


Construction of T for the OLH 0,’ 


Hadamard product of M and §S, adding a row of Os, and the mirror image, 


results in the design. 


A 
=] 


NOaARanNaoPMnaounkwn-aEe 



























































Var 1 Var 2 Var 3 Var 4 | Var5 | Var6 | Var 7 Var 8 
1 -2 -4 -8 3 7 5 -6 
2 1 -3 -7 -4 -8 6 5 
3 -4 2 -6 -1 5 -7 8 
4 3 1 +5 2 -6 -8 -7 
5 -6 -8 4 7 -3 -1 2 
6 5 -7 3 -8 4 -2 -1 
7 -8 6 2 -5 -1 3 -4 
8 7 5 1 6 2 4 3 
0 0 0 0 0 0 0 0 
-1 2 4 8 -3 -7 -5 6 
-2 -1 3 7 4 8 -6 -5 
-3 4 -2 6 1 +5 7 -8 
-4 -3 -1 5 -2 6 8 7 
-5 6 8 -4 -7 3 1 -2 
-6 -5 7 -3 8 -4 2 1 
-7 8 -6 -2 5 | -3 4 
-8 -7 +5 -1 -6 -2 -4 -3 
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APPENDIX B. CATALOGUE OF ORTHOGONAL LATIN 
HYPERCUBE DESIGNS 


The designs that are included are as follows. 


























Description Location 
Design Matrix for 07’ Page 44 
Design Matrix for O,’ Page 44 
Design Matrix for Of? Page 45 
Design Matrix for Of? Page 46 
Other Design Matrix for O%; to | SEED Center Website 
ois http://harvest.nps.edu/ 
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Design Matrix for 0,’ 





xe 


-7 


5 


-8 
4 





x6 


-3 


-8 


-7 


-5 


4 


-2 





x5 


4 


5 


-8 


-7 


-6 





x4 
-3 
-7 
-6 


4 
-8 
-2 





x3 


-8 





x2 


-2 


4 


-6 


-3 


5 


7 








x1 



























































Design Matrix for O,’ 





x8 


7 


4 


5 


-2 





x7 


-8 


-2 


6 


-3 





x6 


-8 


-6 
-3 


4 





xo 


4 


-8 


-2 
7 





x4 


-7 


5 


4 
-3 
2 





x3 


-3 


-7 





x2 


-3 


5 








x1 



































-2 








4 
5 
6 
7 
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Design Matrix for O7? 














































































































x1 x2 x3 x4 xd x6 x7 x8 x9 x10 x11 x12 x13 x14 x15 x16 
1 -2 -4 -8 -16 3 7 15 5 13 9 6 -14 -10 -12 11 
2 1 -3 -7 -15 -4 -8 -16 6 14 10 5 13 9 -11 -12 
3 -4 2 -6 -14 -1 5 13 -7 -15 11 8 16 -12 10 -9 
4 3 1 -5 -13 2 -6 -14 -8 -16 12 -7 -15 11 9 10 
5 -6 -8 4 -12 7 -3 11 -1 9 -13 2 -10 14 16 -15 
6 5 -7 3 -11 -8 4 -12 -2 10 -14 -1 9 -13 15 16 
7 -8 6 2 -10 +5 -1 9 3 -11 -15 -4 12 16 -14 13 
8 7 5 1 -9 6 2 -10 4 -12 -16 3 -11 -15 -13 -14 
9 -10 -12 -16 8 11 15 -7 13 -5 -1 -14 6 2 4 -3 
10 9 -11 -15 7 -12 -16 8 14 -6 -2 13 -5 -1 3 4 
11 -12 10 -14 6 -9 13 5 -15 7 -3 16 -8 4 -2 1 
12 11 9 -13 5 10 -14 6 -16 8 -4 -15 7 -3 -1 -2 
13 -14 -16 12 4 15 -11 -3 -9 -1 5 10 2 6 -8 7 
14 13 -15 11 3 -16 12 4 -10 -2 6 -9 -1 5 -7 -8 
15 -16 14 10 2 -13 -9 -1 11 3 7 -12 -4 -8 6 -5 
16 15 13 9 1 14 10 2 12 4 8 11 3 7 5 6 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
-1 2 4 8 16 -3 -7 -15 5 -13 -9 6 14 10 12 -11 
-2 -1 3 7 15 4 8 16 6 -14 -10 5 -13 -9 11 12 
-3 4 -2 6 14 1 -5 -13 7 15 -11 -8 -16 12 -10 9 
-4 -3 -1 5 13 -2 6 14 8 16 -12 7 15 -11 -9 -10 
-5 6 8 -4 12 -7 3 -11 1 -9 13 -2 10 -14 -16 15 
-6 -5 f. -3 11 8 -4 12 2 -10 14 1 -9 13 -15 -16 
-7 8 6 -2 10 5 1 -9 -3 11 15 4 -12 -16 14 -13 
-8 -7 5 -1 9 -6 -2 10 -4 12 16 -3 11 15 13 14 
-9 10 12 16 -8 -11 -15 7 -13 5 1 14 -6 -2 -4 3 
-10 -9 11 15 -7 12 16 -8 -14 6 2 -13 5 1 -3 -4 
-11 12 -10 14 -6 9 -13 5 15 -7 3 -16 8 -4 2 -1 
-12 -11 -9 13 5 -10 14 6 16 -8 4 15 -7 3 1 2 
-13 14 16 -12 -4 -15 11 3 9 1 -5 -10 -2 6 8 -7 
-14 -13 15 -11 -3 16 -12 -4 10 2 -6 9 1 5 7 8 
-15 16 -14 -10 -2 13 9 1 -11 -3 -7 12 4 8 -6 5 
-16 -15 -13 -9 -1 -14 -10 -2 -12 -4 -8 -11 -3 -7 5 -6 
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Design Matrix for O7? 




















































































































































































































x1 x2 x3 x4 xd x6 x7 x8 x9 x10 x11 X12 x13 x14 x15 x16 
1 -2 -4 -8 -16 -32 3 7 15 31 5 13 29 9 25 17 
2 1 -3 -7 -15 -31 -4 -8 -16 -32 6 14 30 10 26 18 
3 -4 2 -6 -14 -30 -1 5 13 29 -7 -15 -31 11 27 19 
4 3 1 -5 -13 -29 2 6 -14 -30 -8 -16 -32 12 28 20 
5 6 -8 4 -12 -28 vA -3 11 27 -1 9 25 -13 -29 21 
6 5 -7 3 -11 -27 -8 4 -12 -28 -2 10 26 -14 -30 22 
7 -8 6 2 -10 -26 5 -1 9 25 3 -11 -27 -15 -31 23 
8 7 5 1 -9 -25 6 2 -10 -26 4 -12 -28 -16 -32 24 
9 -10 -12 -16 8 -24 11 15 -7 23 13 5 21 -1 17 -25 
10 -11 -15 7 -23 -12 -16 8 -24 14 6 22 -2 18 -26 
11 -12 10 -14 6 -22 -9 13 5 21 -15 7 -23 -3 19 -27 
12 11 9 -13 5 -21 10 -14 6 -22 -16 8 -24 -4 20 -28 
13 -14 -16 12 4 -20 15 -11 -3 19 -9 -1 17 5 -21 -29 
14 13 -15 11 3 -19 -16 12 4 -20 -10 -2 18 6 -22 -30 
15 -16 14 10 2 -18 -13 -9 -1 17 11 3 -19 7 -23 -31 
16 15 13 9 1 -17 14 10 2 -18 12 4 -20 8 -24 -32 
17 -18 -20 -24 -32 16 19 23 31 -15 21 29 -13 25 -9 -1 
18 17 -19 -23 -31 15 -20 -24 -32 16 22 30 -14 26 -10 -2 
19 -20 18 -22 -30 14 -17 21 29 -13 -23 -31 15 27 -11 -3 
20 19 17 -21 -29 13 18 -22 -30 14 -24 -32 16 28 -12 -4 
21 -22 -24 20 -28 12 23 -19 27 -11 -17 25 -9 -29 13 5 
22 21 -23 19 -27 11 -24 20 -28 12 -18 26 -10 -30 14 6 
23 -24 22 18 -26 10 -21 -17 25 -9 19 -27 11 -31 15 -7 
24 23 21 17 -25 9 22 18 -26 10 20 -28 12 -32 16 -8 
25 -26 -28 -32 24 8 27 31 -23 -7 29 -21 5 -17 -1 9 
26 25 -27 -31 23 7 -28 -32 24 8 30 -22 -6 -18 -2 10 
27 -28 26 -30 22 6 -25 29 -21 5 -31 23 7 -19 -3 11 
28 27 25 -29 21 5 26 -30 22 6 -32 24 8 -20 -4 12 
29 -30 -32 28 20 4 31 -27 -19 -3 -25 -17 -1 21 5 13 
30 29 -31 27 19 3 -32 28 20 4 -26 -18 -2 22 6 14 
31 -32 30 26 18 2 -29 -25 -17 -1 27 19 3 23 7 15 
32 31 29 25 17 1 30 26 18 2 28 20 4 24 8 16 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
-1 2 4 8 16 32 -3 -7 -15 -31 5 -13 -29 -9 -25 -17 
-2 -1 3 7 15 31 4 8 16 32 6 -14 -30 -10 -26 -18 
-3 4 -2 6 14 30 1 5 -13 -29 7 15 31 -11 -27 -19 
-4 -3 -1 5 13 29 -2 6 14 30 8 16 32 -12 -28 -20 
5 6 8 -4 12 28 -7 3 -11 -27 1 -9 -25 13 29 -21 
6 5 7 -3 11 27 8 -4 12 28 2 -10 -26 14 30 -22 
-7 8 -6 -2 10 26 5 1 -9 -25 -3 11 27 15 31 -23 
-8 -7 5 -1 9 25 6 -2 10 26 -4 12 28 16 32 -24 
-9 10 12 16 -8 24 -11 -15 7 -23 -13 5 -21 1 -17 25 
-10 -9 11 15 -7 23 12 16 -8 24 -14 6 -22 2 -18 26 
-11 12 -10 14 -6 22 9 -13 5 -21 15 -7 23 3 -19 27 
-12 -11 -9 13 5 21 -10 14 6 22 16 -8 24 4 -20 28 
-13 14 16 -12 -4 20 -15 11 3 -19 9 1 -17 5 21 29 
-14 -13 15 -11 -3 19 16 -12 -4 20 10 2 -18 -6 22 30 
-15 16 -14 -10 -2 18 13 9 1 -17 -11 -3 19 -7 23 31 
-16 -15 -13 -9 -1 17 -14 -10 -2 18 -12 -4 20 -8 24 32 
-17 18 20 24 32 -16 -19 -23 -31 15 -21 -29 13 -25 9 1 
-18 -17 19 23 31 -15 20 24 32 -16 -22 -30 14 -26 10 2 
-19 20 -18 22 30 -14 17 -21 -29 13 23 31 -15 -27 11 3 
-20 -19 -17 21 29 -13 -18 22 30 -14 24 32 -16 -28 12 4 
-21 22 24 -20 28 -12 -23 19 -27 11 17 -25 9 29 -13 5 
-22 -21 23 -19 27 -11 24 -20 28 -12 18 -26 10 30 -14 6 
-23 24 -22 -18 26 -10 21 17 -25 9 -19 27 -11 31 -15 7 
-24 -23 -21 -17 25 -9 -22 -18 26 -10 -20 28 -12 32 -16 8 
-25 26 28 32 -24 -8 -27 -31 23 7 -29 21 5 17 1 -9 
-26 -25 27 31 -23 -7 28 32 -24 -8 -30 22 6 18 2 -10 
-27 28 -26 30 -22 6 25 -29 21 5 31 -23 -7 19 3 -11 
-28 -27 -25 29 -21 -5 -26 30 -22 6 32 -24 -8 20 4 -12 
-29 30 32 -28 -20 -4 -31 27 19 3 25 17 1 -21 5 -13 
-30 -29 31 -27 -19 -3 32 -28 -20 -4 26 18 2 -22 6 -14 
-31 32 -30 -26 -18 -2 29 25 17 1 -27 -19 -3 -23 -7 -15 
-32 -31 -29 -25 -17 -1 -30 -26 -18 -2 -28 -20 -4 -24 -8 -16 





















































APPENDIX C. PROPERTIES OF LATIN HYPERCUBES WITH DIFFERENT CONSTRUCTION 

































































OLH NOLH (SEED) Mean of 100 RLHS Mean of 100 RLHS-F 

k n cond(X™X) cond(X"X) cond(X"X) cond(X™X) 

Pamp | Mm | ML2 | CL2 Pamp | Mm | ML2 | CL2 Pamp | Mm | ML2 | CL2 Pamp | Mm | ML2 | CL2 
7 17 1.00 0.00 1.48 0.17 0.26 1.00 0.00 1.48 0.15 0.24 9.53 0.52 0.97 0.57 0.45 2.24 0.23 1.11 0.19 0.27 
8 17 1.00 0.00 1.79 0.31 0.31 - - - - - 14.41 0.57 1.12 0.92 0.53 2.47 0.24 1.27 0.35 0.33 
11 33 1.00 0.00 1.67 0.95 0.41 1.12 0.02 1.76 0.73 0.36 8.44 0.44 1.28 1.94 0.56 1.76 0.14 1.41 0.83 0.38 
15 33 1.00 0.00 2.20 7.01 0.73 - - - - - 16.64 0.48 1.72 11.31 0.96 2.11 0.17 1.96 6.23 0.71 
16 33 1.00 0.00 2.42 11.52 0.83 - - - - - 19.73 0.49 1.83 17.06 1.08 2.24 0.17 2.06 9.94 0.82 
16 65 1.00 0.00 1.79 7.98 0.72 1.10 0.02 2.04 4.46 0.54 6.47 0.35 1.60 8.61 0.73 1.45 0.08 1.73 4.80 0.57 
22 129 1.00 0.00 1.79 96.58 1.39 1.04 0.01 2.27 37.78 0.86 4.65 0.26 1.97 53.45 1.03 1.27 0.05 2.06 37.26 0.89 
26 65 1.00 0.00 2.70 867.17 2.25 - - - - - 15.50 0.39 2.53 562.75 2.42 1.77 0.10 2.77 431.13 2.08 
29 257 1.00 0.00 1.81 2.E+04 2.92 1.02 0.00 2.73 482.91 1.44 3.61 0.20 2.35 488.63 1.61 1.17 0.03 2.42 392.36 1.48 
31 65 1.00 0.00 3.19 1.E+04 | 3.816 - - - - - 22.84 0.40 2.96 4.E+03 4.33 1.97 0.12 3.26 4.E+03 3.77 
32 65 1.00 0.00 3.34 2.E+04 | 4.261 - - - = = 2 - 7 ~ - = = S : 2 
37 513 1.00 0.00 1.86 5.E+04 6.81 - - - - - - - - - - - - - - - 
42 129 1.00 0.00 3.27 2.E+06 | 11.34 - - - - - 11.26 0.30 3.44 2.E+05 | 10.18 1.54 0.07 3.68 2.E+05 9.32 
46 1025 1.00 0.00 1.91 2.E+06 | 18.24 - - - = = 7 : : 2 = = = s _ . 
57 129 1.00 0.00 4.14 5.E+09 | 49.27 - - - - 2 - E - 2 < S 3 E z 2 
63 129 1.00 0.00 4.56 1.E+11 | 89.03 - - - - - - - - - - - - - - - 
64 129 1.00 0.00 4.67 2.E+11 | 98.99 - - - = 2 < 7 7 = 5 = = 7 a z 
64 257 1.00 0.00 3.88 2.E+11 | 131.73 - - - - 2 = Z - 7 zs 3 = 2 2 & 
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OLH NOLH (SEED) Mean of 100 RLHS Mean of 100 RLHS-F 
k n cond(X™X) cond(X"X) cond(X™X) cond(X™X) 
Pamp | Mm | ML2 | CL2 Pamp | Mm | ML2 | CL2 Pamp | Mm | ML2 | CL2 Pamp | Mm | ML2 | CL2 
93 513 1.00 0.00 4.52 | 1.E+18 | 4.E+03 - - 2 7 2 - 7 - = é s = 2 S & 
99 257 1.00 0.00 5.25 | 3.E+19 | 5.E+03 - - 7 - - 7 7 - = - 2 a 4 S ps 
120 257 1.00 0.00 6.13 | 2.E+24 | 3.6+04 - - . = = 7 - = = : = 2 = d e 
127 257 1.00 0.00 6.49 | 9.E+25 | 6.E+04 - - = zs S - 7 - 2 s S af 4 2 
128 257 1.00 0.00 6.57 | 1.E+26 | 7.E+04 - - 7 - - = 7 a i. 7 2 - s 5 a 
130 | 1025 1.00 0.00 5.12 | 2.6+27 | 5.6+05 - - 2 = = a 7 7 = = = = = - - 
163 513 1.00 0.00 6.52 | 9.E+34 | 9.E+06 - - 2 zi “ - 7 - : E 5 . e = = 
219 513 1.00 0.00 8.05 | 1.E+48 | 2.E+09 - - - - - 7 7 - - : 2 2 = 5 a 
247 513 1.00 0.00 8.91 | 3.E+54 | 2.E+10 - - 2 = = 7 = = 2 = = B _ 7 : 
255 513 1.00 0.00 9.21 | 2.E+56 | 5.E+10 - - = 7 2 - zs - : = = - z 3 2 
256 513 1.00 0.00 9.26 | 3.E+56 | 5.E+10 - - - - - < 7 - 7 a 7 = s 5 a 
256 | 1025 1.00 0.00 7.93 | 7.E+57 | 9.E+11 - - = = = = 7 7 = S = - = 2 2 
382 | 1025 1.00 0.00 10.35 | 5.E+87 | 4.E+17 - - - - 2 - 7 - = é E = = a = 
466 | 1025 1.00 0.00 11.99 |2.E+107 | 9.E+20 - - 7 - - 7 - - 5 _ 7 es = s e 
502 | 1025 1.00 0.00 12.79 |3.E+115 | 2.6+22 - - = = < 7 - = = : = e = Z . 
511 1025 1.00 0.00 13.04 |3.6+117 | 4.6+22 - - 2 . ‘ - 7 - = = Ss 2 2 a a 
512 | 1025 1.00 0.00 13.08 |5.E+117 | 4.6+22 - - 7 - - : 7 - 2 2 Z e = & 2 
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